…g mappings (elastic#188145)

## Summary

Resolves elastic#178528.

Some packages declare `constant_keyword` type fields without an explicit
value. This causes ES to fill in the value in the mappings using the
first ingested value.

When upgrading this type of package & field after the value has already
been populated in this way, the mappings update fail due to pushing a
`null` value into an existing value, triggering unnecessary rollovers.

This PR fixes that by filling in the empty values from the existing
mappings.

## Test
1. On an empty cluster, turn on debug logs
2. Set up Fleet Server policy and Fleet Server agent
3. Force install old version of Elastic Agent integration, v1.19.2:
```
POST kbn:/api/fleet/epm/packages/elastic_agent/1.19.2
{
  "force": true
}
```
4. Create a new empty policy, **deselect system and agent monitoring**
(otherwise the integration will be upgraded, we do not want this yet)
5. Manually add Elastic Agent integration v1.19.2 to the new policy
6. Edit the policy to enable logs and metrics monitoring
7. Enroll agent into the policy, confirm that monitoring logs and
metrics are being ingested and that a value exists for `event.dataset`
mapping for the logs:
```
GET logs-elastic_agent*/_mappings
```
```
            "dataset": {
              "type": "constant_keyword",
              "value": "elastic_agent"
            }
```
9. Upgrade Elastic Agent integration to v1.20.0 (note we are not
upgrading to the newest versions, 2.0+, because these **are** expected
to trigger rollovers for some data streams):
```
POST kbn:/api/fleet/epm/packages/elastic_agent/1.20.0
{
  "force": true
}
```
10. Confirm in Kibana logs that no rollovers triggered during the
upgrade
11. Confirm that there is still only 1 backing index for monitoring
logs:
```
GET logs-elastic_agent*
```

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

(cherry picked from commit b7c96f4)

…elds from existing mappings (#188145) (#188170)

# Backport

This will backport the following commits from `main` to `8.15`:
- [[UII] Fill in empty values for `constant_keyword` fields
from existing mappings
(#188145)](#188145)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Jen
Huang","email":"its.jenetic@gmail.com"},"sourceCommit":{"committedDate":"2024-07-12T03:05:03Z","message":"[UII]
Fill in empty values for `constant_keyword` fields from existing
mappings (#188145)\n\n## Summary\r\n\r\nResolves
https://github.com/elastic/kibana/issues/178528.\r\n\r\nSome packages
declare `constant_keyword` type fields without an explicit\r\nvalue.
This causes ES to fill in the value in the mappings using the\r\nfirst
ingested value.\r\n\r\nWhen upgrading this type of package & field after
the value has already\r\nbeen populated in this way, the mappings update
fail due to pushing a\r\n`null` value into an existing value, triggering
unnecessary rollovers.\r\n\r\nThis PR fixes that by filling in the empty
values from the existing\r\nmappings.\r\n\r\n## Test\r\n1. On an empty
cluster, turn on debug logs\r\n2. Set up Fleet Server policy and Fleet
Server agent\r\n3. Force install old version of Elastic Agent
integration, v1.19.2:\r\n```\r\nPOST
kbn:/api/fleet/epm/packages/elastic_agent/1.19.2\r\n{\r\n \"force\":
true\r\n}\r\n```\r\n4. Create a new empty policy, **deselect system and
agent monitoring**\r\n(otherwise the integration will be upgraded, we do
not want this yet)\r\n5. Manually add Elastic Agent integration v1.19.2
to the new policy\r\n6. Edit the policy to enable logs and metrics
monitoring\r\n7. Enroll agent into the policy, confirm that monitoring
logs and\r\nmetrics are being ingested and that a value exists for
`event.dataset`\r\nmapping for the logs:\r\n```\r\nGET
logs-elastic_agent*/_mappings\r\n```\r\n```\r\n \"dataset\": {\r\n
\"type\": \"constant_keyword\",\r\n \"value\": \"elastic_agent\"\r\n
}\r\n```\r\n9. Upgrade Elastic Agent integration to v1.20.0 (note we are
not\r\nupgrading to the newest versions, 2.0+, because these **are**
expected\r\nto trigger rollovers for some data streams):\r\n```\r\nPOST
kbn:/api/fleet/epm/packages/elastic_agent/1.20.0\r\n{\r\n \"force\":
true\r\n}\r\n```\r\n10. Confirm in Kibana logs that no rollovers
triggered during the\r\nupgrade\r\n11. Confirm that there is still only
1 backing index for monitoring\r\nlogs:\r\n```\r\nGET
logs-elastic_agent*\r\n```\r\n\r\n### Checklist\r\n\r\nDelete any items
that are not applicable to this PR.\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"b7c96f4c09e88b820664bbd0bb996844dd50a0e6","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Fleet","backport:prev-minor","v8.16.0"],"title":"[UII]
Fill in empty values for `constant_keyword` fields from existing
mappings","number":188145,"url":"https://github.com/elastic/kibana/pull/188145","mergeCommit":{"message":"[UII]
Fill in empty values for `constant_keyword` fields from existing
mappings (#188145)\n\n## Summary\r\n\r\nResolves
https://github.com/elastic/kibana/issues/178528.\r\n\r\nSome packages
declare `constant_keyword` type fields without an explicit\r\nvalue.
This causes ES to fill in the value in the mappings using the\r\nfirst
ingested value.\r\n\r\nWhen upgrading this type of package & field after
the value has already\r\nbeen populated in this way, the mappings update
fail due to pushing a\r\n`null` value into an existing value, triggering
unnecessary rollovers.\r\n\r\nThis PR fixes that by filling in the empty
values from the existing\r\nmappings.\r\n\r\n## Test\r\n1. On an empty
cluster, turn on debug logs\r\n2. Set up Fleet Server policy and Fleet
Server agent\r\n3. Force install old version of Elastic Agent
integration, v1.19.2:\r\n```\r\nPOST
kbn:/api/fleet/epm/packages/elastic_agent/1.19.2\r\n{\r\n \"force\":
true\r\n}\r\n```\r\n4. Create a new empty policy, **deselect system and
agent monitoring**\r\n(otherwise the integration will be upgraded, we do
not want this yet)\r\n5. Manually add Elastic Agent integration v1.19.2
to the new policy\r\n6. Edit the policy to enable logs and metrics
monitoring\r\n7. Enroll agent into the policy, confirm that monitoring
logs and\r\nmetrics are being ingested and that a value exists for
`event.dataset`\r\nmapping for the logs:\r\n```\r\nGET
logs-elastic_agent*/_mappings\r\n```\r\n```\r\n \"dataset\": {\r\n
\"type\": \"constant_keyword\",\r\n \"value\": \"elastic_agent\"\r\n
}\r\n```\r\n9. Upgrade Elastic Agent integration to v1.20.0 (note we are
not\r\nupgrading to the newest versions, 2.0+, because these **are**
expected\r\nto trigger rollovers for some data streams):\r\n```\r\nPOST
kbn:/api/fleet/epm/packages/elastic_agent/1.20.0\r\n{\r\n \"force\":
true\r\n}\r\n```\r\n10. Confirm in Kibana logs that no rollovers
triggered during the\r\nupgrade\r\n11. Confirm that there is still only
1 backing index for monitoring\r\nlogs:\r\n```\r\nGET
logs-elastic_agent*\r\n```\r\n\r\n### Checklist\r\n\r\nDelete any items
that are not applicable to this PR.\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"b7c96f4c09e88b820664bbd0bb996844dd50a0e6"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/188145","number":188145,"mergeCommit":{"message":"[UII]
Fill in empty values for `constant_keyword` fields from existing
mappings (#188145)\n\n## Summary\r\n\r\nResolves
https://github.com/elastic/kibana/issues/178528.\r\n\r\nSome packages
declare `constant_keyword` type fields without an explicit\r\nvalue.
This causes ES to fill in the value in the mappings using the\r\nfirst
ingested value.\r\n\r\nWhen upgrading this type of package & field after
the value has already\r\nbeen populated in this way, the mappings update
fail due to pushing a\r\n`null` value into an existing value, triggering
unnecessary rollovers.\r\n\r\nThis PR fixes that by filling in the empty
values from the existing\r\nmappings.\r\n\r\n## Test\r\n1. On an empty
cluster, turn on debug logs\r\n2. Set up Fleet Server policy and Fleet
Server agent\r\n3. Force install old version of Elastic Agent
integration, v1.19.2:\r\n```\r\nPOST
kbn:/api/fleet/epm/packages/elastic_agent/1.19.2\r\n{\r\n \"force\":
true\r\n}\r\n```\r\n4. Create a new empty policy, **deselect system and
agent monitoring**\r\n(otherwise the integration will be upgraded, we do
not want this yet)\r\n5. Manually add Elastic Agent integration v1.19.2
to the new policy\r\n6. Edit the policy to enable logs and metrics
monitoring\r\n7. Enroll agent into the policy, confirm that monitoring
logs and\r\nmetrics are being ingested and that a value exists for
`event.dataset`\r\nmapping for the logs:\r\n```\r\nGET
logs-elastic_agent*/_mappings\r\n```\r\n```\r\n \"dataset\": {\r\n
\"type\": \"constant_keyword\",\r\n \"value\": \"elastic_agent\"\r\n
}\r\n```\r\n9. Upgrade Elastic Agent integration to v1.20.0 (note we are
not\r\nupgrading to the newest versions, 2.0+, because these **are**
expected\r\nto trigger rollovers for some data streams):\r\n```\r\nPOST
kbn:/api/fleet/epm/packages/elastic_agent/1.20.0\r\n{\r\n \"force\":
true\r\n}\r\n```\r\n10. Confirm in Kibana logs that no rollovers
triggered during the\r\nupgrade\r\n11. Confirm that there is still only
1 backing index for monitoring\r\nlogs:\r\n```\r\nGET
logs-elastic_agent*\r\n```\r\n\r\n### Checklist\r\n\r\nDelete any items
that are not applicable to this PR.\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"b7c96f4c09e88b820664bbd0bb996844dd50a0e6"}}]}]
BACKPORT-->

Co-authored-by: Jen Huang <its.jenetic@gmail.com>

## Summary

Follow up to #188145. In some edge cases
(elastic/sdh-beats#5156), users could override
the index template used by integration data streams. It is possible to
create an index template without mappings, this causes
`fillConstantKeywordValues` to receive an undefined object when
upgrading the integration, and the upgrade then fails.

This PR makes the backfill operation here more fail-safe.

## Summary

Follow up to elastic#188145. In some edge cases
(elastic/sdh-beats#5156), users could override
the index template used by integration data streams. It is possible to
create an index template without mappings, this causes
`fillConstantKeywordValues` to receive an undefined object when
upgrading the integration, and the upgrade then fails.

This PR makes the backfill operation here more fail-safe.

(cherry picked from commit b5abc4e)

## Summary

Follow up to elastic#188145. In some edge cases
(elastic/sdh-beats#5156), users could override
the index template used by integration data streams. It is possible to
create an index template without mappings, this causes
`fillConstantKeywordValues` to receive an undefined object when
upgrading the integration, and the upgrade then fails.

This PR makes the backfill operation here more fail-safe.

(cherry picked from commit b5abc4e)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[UII] Make constant keyword backfill optional
(#192921)](#192921)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Jen
Huang","email":"its.jenetic@gmail.com"},"sourceCommit":{"committedDate":"2024-09-14T02:34:46Z","message":"[UII]
Make constant keyword backfill optional (#192921)\n\n##
Summary\r\n\r\nFollow up to #188145. In some edge
cases\r\n(elastic/sdh-beats#5156), users could
override\r\nthe index template used by integration data streams. It is
possible to\r\ncreate an index template without mappings, this
causes\r\n`fillConstantKeywordValues` to receive an undefined object
when\r\nupgrading the integration, and the upgrade then
fails.\r\n\r\nThis PR makes the backfill operation here more
fail-safe.","sha":"b5abc4ec7e308815b0f338f4c836a9caf3ee48a3","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Fleet","v9.0.0","backport:prev-major"],"title":"[UII]
Make constant keyword backfill
optional","number":192921,"url":"https://github.com/elastic/kibana/pull/192921","mergeCommit":{"message":"[UII]
Make constant keyword backfill optional (#192921)\n\n##
Summary\r\n\r\nFollow up to #188145. In some edge
cases\r\n(elastic/sdh-beats#5156), users could
override\r\nthe index template used by integration data streams. It is
possible to\r\ncreate an index template without mappings, this
causes\r\n`fillConstantKeywordValues` to receive an undefined object
when\r\nupgrading the integration, and the upgrade then
fails.\r\n\r\nThis PR makes the backfill operation here more
fail-safe.","sha":"b5abc4ec7e308815b0f338f4c836a9caf3ee48a3"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192921","number":192921,"mergeCommit":{"message":"[UII]
Make constant keyword backfill optional (#192921)\n\n##
Summary\r\n\r\nFollow up to #188145. In some edge
cases\r\n(elastic/sdh-beats#5156), users could
override\r\nthe index template used by integration data streams. It is
possible to\r\ncreate an index template without mappings, this
causes\r\n`fillConstantKeywordValues` to receive an undefined object
when\r\nupgrading the integration, and the upgrade then
fails.\r\n\r\nThis PR makes the backfill operation here more
fail-safe.","sha":"b5abc4ec7e308815b0f338f4c836a9caf3ee48a3"}}]}]
BACKPORT-->

Co-authored-by: Jen Huang <its.jenetic@gmail.com>

# Backport

This will backport the following commits from `main` to `8.15`:
- [[UII] Make constant keyword backfill optional
(#192921)](#192921)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Jen
Huang","email":"its.jenetic@gmail.com"},"sourceCommit":{"committedDate":"2024-09-14T02:34:46Z","message":"[UII]
Make constant keyword backfill optional (#192921)\n\n##
Summary\r\n\r\nFollow up to #188145. In some edge
cases\r\n(elastic/sdh-beats#5156), users could
override\r\nthe index template used by integration data streams. It is
possible to\r\ncreate an index template without mappings, this
causes\r\n`fillConstantKeywordValues` to receive an undefined object
when\r\nupgrading the integration, and the upgrade then
fails.\r\n\r\nThis PR makes the backfill operation here more
fail-safe.","sha":"b5abc4ec7e308815b0f338f4c836a9caf3ee48a3","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Fleet","v9.0.0","backport:prev-major"],"title":"[UII]
Make constant keyword backfill
optional","number":192921,"url":"https://github.com/elastic/kibana/pull/192921","mergeCommit":{"message":"[UII]
Make constant keyword backfill optional (#192921)\n\n##
Summary\r\n\r\nFollow up to #188145. In some edge
cases\r\n(elastic/sdh-beats#5156), users could
override\r\nthe index template used by integration data streams. It is
possible to\r\ncreate an index template without mappings, this
causes\r\n`fillConstantKeywordValues` to receive an undefined object
when\r\nupgrading the integration, and the upgrade then
fails.\r\n\r\nThis PR makes the backfill operation here more
fail-safe.","sha":"b5abc4ec7e308815b0f338f4c836a9caf3ee48a3"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192921","number":192921,"mergeCommit":{"message":"[UII]
Make constant keyword backfill optional (#192921)\n\n##
Summary\r\n\r\nFollow up to #188145. In some edge
cases\r\n(elastic/sdh-beats#5156), users could
override\r\nthe index template used by integration data streams. It is
possible to\r\ncreate an index template without mappings, this
causes\r\n`fillConstantKeywordValues` to receive an undefined object
when\r\nupgrading the integration, and the upgrade then
fails.\r\n\r\nThis PR makes the backfill operation here more
fail-safe.","sha":"b5abc4ec7e308815b0f338f4c836a9caf3ee48a3"}}]}]
BACKPORT-->

Co-authored-by: Jen Huang <its.jenetic@gmail.com>